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SYSTEMS AND METHODS FOR SELECTING SURVEY 
QUESTIONS AND AVAILABLE RESPONSES 



Background of the Invention 

[0001] The present invention relates to systems and 
5 methods for selecting survey questions and available 
responses to provide to survey participants. More 
particularly, the present invention relates to systems 
and methods for incorporating multiple survey questions 
that cover various categories and lists of available 

10 responses into easily readable, brief questionnaires. 
[0002] Surveys are typically conducted by an 
organization in person, over the phone, or via the 
World Wide Web. There may be various reasons for 
organizations, such as retail companies, to administer 

15 surveys. For example, surveys are a personable and 
effective way for receiving accurate feedback from 
existing customers in order to provide these customers 
with more commercial opportunities, better service, 
etc. Surveys may also provide an organization with 

20 insight as to the behavior of survey participants and 
the goods and services they consume. Surveys may also 
provide detailed demographic or ideological makeup of a 
group of people . 
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[0003] In a physical store, on a telephone, or via 
the World Wide Web, it is impractical to require a 
survey participant to partake in a survey that lasts 
more than a few minutes. However, in order to make 
5 accurate assessments about a large number of categories 
projected over a large number of individuals, it is 
preferable for an organization to administer long and 
highly detailed surveys. Indeed, there exists a 
significant trade-off between the burden an 

10 organization may want to impose on a surveyed 

individual and the quality and quantity of information 
that the individual may be able to provide in a shorter 
period of time. Generally, this means that surveys 
must be limited in the number of questions and issues 

15 that can be addressed. 

[0004] It would therefore be desirable to allow 
organizations to administer surveys that are able to 
efficiently collect survey responses to a large number 
of questions by selectively providing relatively few 

20 survey questions to a large number of individuals. 

[0005] It would also be desirable to selectively 

draw the survey questions asked from a much larger list 
of stored survey questions to optimize the information 
obtained from each surveyed individual in the shortest 

25 amount of time. 

[0006] It would also be desirable to accommodate 

survey questions with very long response lists, while 
keeping the presentation of individual survey questions 
relatively short. 

30 [0007] In some cases, a survey question may be 

provided that does not have an appropriate response 
that a survey participant would be willing to select. 
In such cases, a "None of these" response option may 
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provide the survey participant with an opportunity to 
indicate that none of the responses provided are 
adequate for responding to the question. In some cases 
the survey responses provided to the survey participant 
5 may be a non- inclusive list of many stored available 
survey responses (e.g., when at least one available 
survey response to a survey question is hidden from the 
survey participant) . In such cases, the survey 
participant may have selected one of the hidden survey 

10 responses instead of the "None of these" option had 
that hidden response actually been available to the 
participant. Systems and methods are publicly known 
that estimate how "None of these" responses may have 
been redistributed among the hidden survey responses 

15 based on the previous selection rate of each response. 
However, these systems and methods do not account for 
the probability that a survey participant would have 
still selected the "None of these" response even if all 
other available responses were provided. 

20 [0008] It is therefore desirable to more accurately 

estimate the rate at which survey responses are 
selected. 

Summary of the Invention 

25 [0009] It is therefore an object of the present 

invention to provide systems and methods for allowing 
organizations to administer surveys that are able to 
efficiently collect survey responses to a large number 
of questions by selectively providing relatively few 

30 survey questions to a large number of individuals. 

[0010] It is also an object of the present invention 
to provide systems and methods for selectively drawing 
the survey questions asked from a larger survey 
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question database to optimize the information obtained 
from each surveyed individual in the shortest amount of 
time . 

[0011] It is also an object of the present invention 

5 to provide systems and methods for accommodating survey 
questions with very long response lists, while keeping 
the presentation of individual survey questions 
relatively short. 

[0012] It is also an object of the present invention 

10 to provide systems and methods for estimating the rate 
at which survey responses are selected. 
[0013] These and other objects of the present 
invention are accomplished by implementing systems and 
methods that are able to incorporate multiple questions 

15 and available responses into easily readable, brief 

survey questionnaires. A variety of survey questions, 
spanning various categories or types of questions may 
be stored in a storage device . Surveys having at least 
one of the stored questions may be provided to users of 

20 a survey software application (e.g., survey 

participants) . The survey application may be 
implemented on any suitable computing device, which may 
be directly accessed by the survey participant or 
administered via a telephone interview or other means. 

2 5 The questions provided in each survey may vary across 
different survey participants to more efficiently 
utilize each individual participants time. 
[0014] The survey application may select survey 

questions to be included in a survey based upon an 

30 inclusion value. The inclusion value for each question 
may be initially programmed into the survey application 
or an initial value may be associated with a data 
record for each question as it is stored. Inclusion 
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values for each survey question may be updated in real- 
time and may be determined based on, for example, 
conditional branching logic programmed into the system, 
the response variance of a survey question, a global 
5 inclusion value multiplier, any suitable criteria for 
determining the inclusion value, or a combination 
thereof. Survey questions with higher inclusion values 
may be selected by the survey application for inclusion 
in a survey more frequently than survey questions with 

10 lower inclusion values. In some arrangements, a 

threshold inclusion value may be designated by the 
survey application. Survey questions having survey 
inclusion values higher than the designated threshold 
inclusion value may be selected for inclusion in the 

15 survey. Survey questions having inclusion values lower 
than the designated threshold inclusion value may be 
excluded from the survey. 

[0015] The systems and methods of the present 

invention may also determine the rate at which survey 

20 responses have been selected. Frequently selected 
responses may be presented less often in subsequent 
surveys, thereby increasing the sample size on the 
remaining responses. A survey response's selection 
rate indicates the percent of the time that each survey 

25 response is selected and may be calculated based on the 
number of times a response is selected divided by the 
number of times it was presented. Questions with long 
lists of responses may, therefore, be presented to a 
survey participant in a brief and manageable format by 

3 0 reducing the number of responses provided based on the 
response selection rate. 

[0016] Moreover, because the presentation of each 
question in a survey may include only a limited number 
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of responses to choose from, it may be desirable for 
each response list to include a "fallback" response, 
for example "None of the above," "None of these," or 
any other suitable response that allows a survey 
5 participant to indicate that none of the responses 
provided were adequate for responding to the survey 
question presented. The fallback responses may be 
categorized as "Real" or "Reallocated" selections. 
Real selections are survey response selections that 

10 would have remained the same even if all available 
responses were presented to a survey participant. 
Reallocated selections are survey response selections 
that may have been different had survey responses not 
provided in a non- inclusive list of responses been 

15 presented to the survey participant. Reallocated 
selections may be distributed among other response 
selections according to the distribution of the survey 
response selection rate and the probability that a 
survey participant would have selected a hidden 

2 0 response had it been provided. This probability may be 

estimated in a number of ways. For example, 
reallocated selections (e.g., adders) for each response 
may be determined based on the selection rate of each 
response and the probability that a survey 
25 participant's response would have been different had 
hidden survey response selections been provided. The 
adders may be used to reallocate "None of these" 
selections . 

3 0 Brief Description of the Drawings 

[0017] The above and other objects and advantages of 
the invention will be more apparent upon consideration 
of the following detailed description, taken in 



conjunction with the accompanying drawings, in which 
like reference numerals refer to like parts throughout, 
and in which: 

[0018] FIG. 1 is a diagram of illustrative software 
and hardware used in accordance with the present 
invention; 

[0019] FIGS. 2 and 3 show illustrative tables for 

storing survey questions, survey responses, and survey 
response information in accordance with the present 
invention; 

[0020] FIGS. 4 and 5 show illustrative display 

screens for providing survey participants with selected 
survey questions and selected survey responses in 
accordance with the present invention; 
[0021] FIGS. 6 and 7 are illustrative output 
displays that show calculations based on stored survey 
response information in accordance with the present 
invention; 

[0022] FIG . 8 is a flow chart of illustrative steps 

involved in selecting survey questions to be provided 
in accordance with the present invention; and 

[0023] FIG. 9 is a flow chart of illustrative steps 

involved in selecting survey responses to be provided 
in accordance with the present invention. 

Detailed Description 

[0024] The present invention is now described in 
more detail in conjunction with FIGS. 1-9. Although 
the present invention may be illustrated as being 
implemented in a retail environment for assisting in 
the administration of retail survey questionnaires, it 
will be understood that the present invention may be 
implemented in other types of environments. For 
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number of multi-tiered databases for storing survey- 
questions, survey response, survey response information 
provided by survey participants, or any other suitable 
information. In some embodiments, not shown, 
5 database 10 6 may be implemented as part of computing 
device 102, or part or all of database 106 may be 
implemented to both computing device 102 and 
application server 108. 

[0028] In FIG. 1, survey application 104 is 

10 implemented on computing device 102 while database 10 6 
is implemented on application server 108. It will be 
understood, however, that the software application (s) 
used in connection with the present invention may be 
implemented by any device included as part of hardware 

15 and software 100 and that the single embodiment of 

FIG. 1 is used merely as an illustration. For example, 
in one embodiment, such as the case of a perfectly 
distributed network (e.g., a thin-client computing 
arrangement, an application service provider 

20 arrangement, etc.), as is typical in a retail 

environment having point-of-sale terminals, all 
software applications may be implemented by application 
server 108, or any other suitable device (e.g., a 
mainframe computer, a supercomputer, etc.), while 

25 personal computing device 102 may only include a user 

interface (e.g., a user input device, a display screen, 
etc . ) . 

[0029] The information in database 106 may include 
survey questions, responses, and survey response 
30 information. The information in database 106 may be in 
any suitable data management format, environment, or 
application. For example, a relational database 
format, an object oriented database format, a data 
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warehouse, a data directory, a knowledge management 
system, or any other suitable format, environment or 
application may be used for storing and indexing 
related information. The hardware (e.g., application 
5 server 108, computing devices 102, etc.) and software 
(e.g., database 106, survey application 104, etc.) may 
use various other hardware and software for making the 
calculations described herein. 

[0030] Database 106 may reside locally (e.g., as 

10 part of or adjacent to computing device 102) , or at a 
location remote from computing device 102 and accessed 
via network 110. Computing device 102 may be coupled 
to network 110 via communications paths 125-127. 
Network 110 may be a local or wide area network (e.g., 

15 the Internet, an intranet, a virtual private network, 
etc.) and may support any combination of wired, 
wireless, or optical communications. Application 
server 10 8 may be coupled to network 110 via 
communications path 129. 

2 0 [0 031] The hardware and software configuration of 
FIG. 1 may also include information sources 12 0 
and 122, which may be a web server, a database, or any 
other suitable device for storing information such as 
an organization's financial information, transaction 

25 information derived from point-of-sale information, 

survey participant profile information, survey response 
information, national economic and industry 
information, or any other suitable information. 
Information sources 120 and 122 may be coupled to 

30 network 110 via communications paths 128 and 130. In 
addition or alternatively, information sources 120 
and 122 may be coupled directly to application 
server 108 via communications paths 131 and 132. In 



either embodiment, the information stored in 
information sources 120 and 122 may be accessed by 
application server 108 or computing device 102. 
[0032] Communications paths 125-132 may be any 
suitable wired or wireless communications path. For 
example, if wire-based, communications paths 125-132 
may be serial connections, parallel connections, 
telephone cables, copper wire, electric cable, fiber 
optic cable, coaxial cable, Ethernet cable, USB cable, 
FireWire cable, component video cables, composite 
cables, any other suitable wire-based communications 
path, or any combination thereof. If wireless, any 
suitable communications protocol or standard such as 
IEEE 802.11, wireless application protocol (WAP), radio 
frequency (RF) , Bluetooth, (Extended) time division 
multiple access (TDMA) , code-division multiple access 
(CDMA) , global systems for mobile communications (GSM) , 
or any other suitable wireless communications path or 
protocol may be used. A combination of wired and 
wireless communication paths may also be used. 
Communications paths 12 5-132 may provide access to 
network 110 via a web server, a network gateway, any 
other suitable device, or a combination thereof. 
[0033] The software and hardware illustrated in 
FIG. 1 may be used to implement the systems and methods 
of the present invention. For example, a survey 
participant may operate computing device 102 to access 
survey application 104. Survey application 104 may 
include any software application that provides survey 
participants with survey questions and available 
responses. Survey application 104 may use information 
in database 106 to create surveys for survey 
participants. For example, survey application 104 may 



- 12 - 



use data stored in database 106 to create display 
screens of survey questions and responses. 
[0034] The data used by survey application 104 to 
generate surveys may be stored in any suitable format . 
FIG. 2 shows an illustrative example of how survey 
questions may be stored in database 106. FIG. 2 
includes survey questions 202-204, which may include 
responses 208-214. It will be understood that it is 
preferable for many survey questions to be stored. 
However, for the purposes of brevity and clarity, only 
several instances of stored survey questions are 
depicted in FIG. 2. The variance 215 and inclusion 
value 216 for each survey question may also be included 
for each survey question stored. 

[0035] FIG. 3 shows an illustrative example of how 

survey responses may be stored in database 106. FIG. 3 
includes a list of survey responses 302-308 that 
correspond to an identified survey question 310. The 
response text 3 09 may also be provided. In the example 
provided in FIG. 3, the identified survey question 310 
(e.g., survey question 1) corresponds to survey 
question 202 of FIG. 2 and the survey responses 302-308 
(e.g., survey responses A-G) correspond to those 
indicated in FIG. 2. The selection rate 311 of each 
stored survey response may also be provided. 
[0036] Survey application 104 may create surveys by 

selecting at least one survey question from a list of 
survey questions 202-204 stored in database 106. 
Survey application 104 may provide a list of survey 
responses 208-214 for each survey question provided. 
The survey responses provided by survey application 104 
may be selected from a list of survey responses (e.g., 
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survey responses 302-308 of FIG. 3) stored in 
database 106 that correspond to each survey question. 
[0037] Survey questions and responses may be 
selected by survey application 104 in real-time 
5 according to a specific criteria, and the survey 
questions may be provided to at least one survey 
participant. The networked arrangement of computing 
devices 102, survey application 104, and database 106 
shown in FIG. 1 allows survey application 104 to 

10 determine whether a survey question meets a given 
criteria and then to provide the selected survey 
question in real-time. While collecting survey 
response information from a survey participant, a real- 
time determination of a survey question's response 

15 variance may be determined. The response variance may 
be updated when a survey participant submits a response 
for a single survey question or at the end of a survey 
consisting of multiple survey questions. The survey 
question responses selected by survey participants 

2 0 (e.g., survey response information) may be transmitted 

to, and stored by, database 106 or any other suitable 
storage device (e.g., information sources 120 and 122, 
etc.). The storage device used to store survey 
response information may include hardware and software 
25 for calculating the response variance for each survey 
question for which responses have been provided by 
survey participants. A survey question's response 
variance may be calculated using known methods for 
calculating the variance (e.g., by determining the 

3 0 square of the standard deviation of responses to a 

particular survey question) . 

[0038] For each survey question stored in 

database 106, there may be a data item corresponding to 
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the response variance for that survey question (e.g., 
variance field 215 of FIG. 2) . Survey application 104 
may utilize this data item when determining which 
survey questions to provide to a survey participant. 
For example, survey application 104 may select survey 
questions having a higher response variance over survey 
questions having a lower response variance in order to 
shorten the survey's duration and to include only those 
questions for which larger sample sizes are needed. 
Survey questions that have low variance may have a 
common response selected by the vast majority of survey 
participants and such questions may, therefore, require 
little further sampling. Accordingly, it may not be 
necessary to include such questions in surveys as 
frequently as those questions with a higher response 
variance. This method of selecting survey questions 
reduces the number of survey questions that a survey 
participant must respond to and, therefore, reduces the 
amount of time needed to administer a survey. 
[0039] A designated variance level may be initially 
determined (e.g., by survey application 104) so that 
all questions are presented with some minimal 
frequency, regardless of the response variance of any 
given question. In a survey presented over a long 
period of time, the variance analysis may be divided 
into time segments and the variance for a more recent 
time period may be calculated independently from prior 
time periods. Therefore, changes in population or 
public sentiment may quickly be detected from survey 
participants 1 responses. For example, survey 
application 104 may designate a particular time period 
and use survey response information (which may include 
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a time stamp) to calculate the response variance for 
the designated time period. 

[0040] In another suitable approach, survey 
application 104 may select survey questions to be 
included in a survey based upon inclusion values. The 
inclusion value for each question may be initially 
programmed into the survey application or an initial 
inclusion value may be associated with a data record 
for each question as it is stored. Inclusion values 
for each survey question may be updated in real-time 
and may be determined based on, for example, 
conditional branching logic programmed into the system, 
the response variance of a survey question, a global 
inclusion value multiplier, any suitable criteria for 
determining the inclusion value, or a combination 
thereof. Survey questions having higher inclusion 
values may be selected by survey application 104 for 
inclusion in a survey more frequently than survey 
questions having lower inclusions values. In some 
arrangements, a threshold inclusion value may be 
designated by the survey application. Survey questions 
having survey inclusion values higher than the 
designated threshold inclusion value may be selected 
for inclusions in the survey. Survey questions having 
inclusion values lower than the designated threshold 
inclusion value may be excluded from the survey. 
[0 041] There are several approaches for determining 
the inclusion value of a survey question. In one 
suitable approach, predefined conditional branching 
logic, which may logically relate survey questions to 
one another based on a survey participant's response, 
may be used to determine the inclusion value of 
subsequent survey questions once the survey participant 
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has provided a response to a survey question. For 
example, one survey participant may be provided with a 
string of survey questions that are different from 
those provided to another survey participant because 
5 the survey participant's responses were different for 
at least one of the questions provided. The 
predetermined conditional branching logic may be 
associated with the stored survey questions or may be 
based on logic programmed into survey application 104 . 

10 [0 042] In another suitable approach, each survey 

question ! s response variance may be used to determine 
the inclusion value of each survey question. For 
example, fluctuations in a given question's response 
variance may be detected if survey participants* 

15 responses are notably inconsistent with responses 

previously provided by other survey participants. A 
desired variance may have been predefined and 
previously programmed into survey application 104 and a 
real-time variance estimate may be compared to the 

20 desired variance. If the variance is found to be above 
the desired variance, the inclusion value may be 
increased for that question and, thus, survey 
application 104 may select that question more often to 
increase the survey question's sample size. 

2 5 Conversely, if the response variance for a survey 

question is below the desired variance programmed into 
survey application 104, the inclusion value may be 
decreased for that question and survey application 104 
may select that survey question less frequently. 

30 [0043] A survey question's inclusion value may be 

related to an initial inclusion value (e.g., designated 
by survey application 104 prior to providing any survey 
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questions), the survey question's variance, and a 
desired variance. For example: 

Where : 

I = Inclusion value 
XBase — Initial inclusion value 
Vcurrent = Current variance 
Vdes = Desired Variance 

5 [0 044] Another suitable approach for determining a 
question's inclusion value may include monitoring 
survey duration and increasing or decreasing a global 
inclusion value multiplier for the questions that have 
not yet been presented. For example, if the survey is 

10 taking a long time for a survey participant to 

complete, the global inclusion value multiplier may be 
decreased in order to reduce the probability of 
inclusion of the remaining survey questions. 
Conversely, if a survey is being completed more 

15 rapidly, the global inclusion value multiplier may be 
increased for each remaining survey question. If 
questions are being selected based on whether their 
inclusion value is higher or lower than a value 
designated by survey application 104, globally 

2 0 increasing or decreasing the inclusion values of the 
remaining questions will increase or decrease the 
number of survey questions provided. 

[0045] The inclusion value may be updated at the 

individual and global level, such that an individual 
2 5 survey may increase or decrease the inclusion value for 
a given question to meet a specified time goal. 
Subsequent surveys may implement global increases or 



v 

j j a|e Current 

1 ~~ 1 Base jr 

V des 
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decreases in inclusion value to average the survey time 
calculated over several recent surveys (e.g., to arrive 
at an average consistent with a predefined average 
programmed into survey application 104) . 
5 Alternatively, the survey may end after a predetermined 
time period. 

[0046] It will be understood that the inclusion 
value of a given survey question may be determined 
using any of the foregoing techniques. For example, 

10 the inclusion value may be determined based on 

predefined conditional branching logic, response 
variance, a global inclusion value multiplier, any 
other suitable information, or any combination thereof. 
In some arrangements, it may be desirable to use a 

15 combination of these techniques to determine a survey 
question's inclusion value, and the various techniques 
may be weighted to arrive at a desired technique for 
selecting survey questions based on inclusion values. 
It is preferred, however, to apply an identical 

2 0 inclusion value calculation to all stored survey 
questions . 

[0047] The foregoing description illustrates systems 
and methods for selecting survey questions. In another 
aspect of the present invention, systems and methods 

25 are provided for selecting survey responses. A survey 
question selected by survey application 104 may include 
many possible survey responses. For example, in 
FIG. 2, survey questions 2 02-204 may include survey 
responses 208-214 (e.g., identified as survey responses 

30 A-U) . Survey responses for survey question 202 (e.g., 
responses A-G) may correspond to survey responses 3 02- 
308 of FIG. 3. A response's selection rate may be 
calculated once the survey participants 1 responses are 
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stored (e.g., using the survey response information). 
A response's selection rate may be calculated based on 
the number of times a survey response was selected by 
survey participants divided by the number of times it 
5 is presented in a survey. 

[0048] For each survey response stored in 
database 106, there may be a data item corresponding to 
the rate at which that survey option is selected by 
survey participants (e.g., selection rate field 311 of 

10 FIG. 3) . Survey application 104 may utilize this data 
item when determining which responses to provide a 
survey participant. Survey application 104 may select 
responses with lower selection rates in order to 
increase the sample size of these survey responses 

15 (e.g., survey responses 302, 303 and 304 may have the 

lowest selection rate of the survey responses available 
for survey question 2 02) . Reducing the number of 
responses presented for each question allows questions 
with long lists of responses to be presented to a 

20 survey participant in a brief and manageable format. 

Also, it may be desirable to select survey responses in 
this manner if the sample size for a survey response 
increases to the point that it is no longer desirable 
to sample the response (e.g., if the sample size is 

25 large enough to account for selection variance of the 

response) . A survey response with a large sample size 
may be displayed less frequently, allowing other survey 
responses to be presented with higher frequency. 
[004 9] It will be understood that a survey 

30 response's selection rate may be determined using the 
aforementioned method regardless of whether a given 
survey question allows for more than one survey 
response to be selected by a survey participant. 
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[0050] Survey questions and responses may be 
presented to a survey participant using, for example, 
survey application 104 and the associated software and 
hardware of FIG. 1. Survey application 104 may present 
5 a series of interactive display screens that may ask 
survey participants to respond to questions and input 
information. Survey application 104 may be implemented 
in conjunction with a standard web browser application 
(e.g., Microsoft's Internet Explorer, Netscape 

10 Navigator, etc.) and accessed over the Internet. 
Alternatively, the survey application may be 
implemented in conjunction with a proprietary or 
commercially available software application interface 
and accessed over a public or private network (e.g., in 

15 an arrangement using a point-of-sale terminal) or a 
proprietary or commercially available software 
application interface accessed locally (e.g., in an 
arrangement using an exit kiosk as computing 
device 102) . Any other suitable arrangement or 

2 0 implementation may also be used. User input devices, 
such as a mouse, keyboard, telephone keypad, touch- 
sensitive display screen, or any other suitable user 
input device, may also be used to allow survey 
participants to interact with survey application 104 

25 and may be included as part of computing device 102. 

Display devices such as a computer monitor, a handheld 
display, or any other suitable device, may be used to 
present display screens of survey questions to the 
survey participant. 

30 [0051] It will be understood that some arrangements 

of the present invention, for example in an arrangement 
using a telephone system to implement survey 
application 104, display screens may not be needed and 
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the selected survey questions and responses may be 
provided audibly over a telephone. In such an 
arrangement, survey participants may indicate responses 
using a telephone keypad and the survey response 
5 information may be transmitted to and stored by the 

storage device, and used to select survey questions and 
responses, in the same manner set forth in the forgoing 
description . 

[0052] Illustrative display screens of survey 

10 questions and responses that may be displayed by survey 
application 104 in an arrangement using, for example, a 
personal computer or kiosk, are set forth in FIGS. 4 
and 5. FIG. 4 shows display screen 400 that may 
include survey question 202. Survey question 202 may 

15 be selected from among many survey questions stored in 
a storage device (e.g., application server 108 of 
FIG. 1) . Survey application 104 may have selected 
survey question 2 02 over other available survey 
questions (e.g., survey questions 203 and 204) to be 

20 presented in display screen 400. For example, FIG. 2 

shows that survey question 2 02 has the highest variance 
215. Survey application 104 may therefore determine 
that responses to survey question 2 02 vary more than 
the responses to other available survey questions. 

25 Alternatively, survey application 104 may have selected 
question 202 because the question's inclusion value 215 
(FIG. 2) is higher than the other available survey 
questions (e.g. survey questions 203 and 204). 
[0053] Responses to survey question 202 may also be 

30 included as part of display screen 400, for example, 
survey responses 3 02, 3 03, 3 04, and 3 08. For each 
survey response provided, there may be an area 4 08 to 
be used by a survey participant to indicate a response. 
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The responses provided in display screen 4 00 may not be 
a complete list of the survey responses available for 
survey question 202. For example, as shown in FIGS. 2 
and 3, survey question 202 may have several possible 
5 survey responses 208-214 (which correspond to 

responses 302-308 of FIG. 3) . However, to provide 
survey responses in a brief and manageable format, only 
several survey responses are provided on display 
screen 400. Once the survey participant has indicated 

10 a response to the survey question, the survey response 
information may be transmitted to and stored by a 
storage device (e.g., application server 108 of FIG. 1 
or any other suitable device) . Survey response 
information may also be stored locally on computing 

15 " device 102 or any other suitable local storage device. 
[0054] FIG. 5 shows display screen 500 which may 
include another survey question, such as survey 
question 204. Survey application 104 may select survey 
question 204 based on the question 1 s inclusion value 

20 (e.g., as shown in inclusion value field 216 of 

FIG. 2) . The inclusion value for survey question 204 
may have been increased in real-time once the survey 
participant selected "Location" survey response 3 02 in 
response to survey question 2 02 (provided in FIG. 4) . 

25 The increase in the inclusion value of question 204 may 
have been caused by conditional branching logic 
associated with that response programmed to 
automatically increase the inclusion value for survey 
question 204 if response 302 was selected for 

30 question 204 (e.g., because survey response 302 and 
survey question 204 both relate to location) . 
[0055] Each survey question selected by survey 
application 104 may include a "None of these" survey 
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response, such as survey responses 308 (FIG. 4) and 508 
(FIG. 5) , or any similar response that allows survey 
participants to indicate that none of the survey 
responses provided offers an adequate response to the 
5 survey question provided. The responses submitted by 
survey participants (e.g., survey response information) 
may be stored in a storage device, such as application 
server 108 or any other suitable device. The stored 
survey response information for each survey question 

10 may be displayed in a table such as table 600 of 
FIG. 6. The presentation count 602, selection 
count 604, and selection rate 606 for each available 
survey response may be indicated. The presentation 
count 602, selection count 604, and selection rate 606 

15 may also be provided for "None of these" response 608. 
[0056] In FIG. 6, the "None of these" response was 

presented 100 times and selected 50 times. Because 
this response may be displayed every time a survey 
question is presented, it may be determined that the 

20 given survey question was presented 100 times. It may 
also be determined, therefore, that the "None of these" 
response was selected half the times the question was 
presented. The average number of survey responses 
presented for each survey question may also be 

25 calculated, for example by totaling the number of times 
all responses were presented and dividing by the number 
of times the survey question was provided. 
[0057] It may be desirable to determine the 
probability that a survey participant's "None of these" 

3 0 results would have been different had the entire list 
or survey questions been provided. The average number 
of survey responses provided for a given survey 
question ("X") may determine the amount of space needed 
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to present the survey question and list of responses. 
For example, a wordy survey question may have room only 
for three responses on a single display screen (and 
therefore present an average of 3 responses each time 
5 the survey question is provided) . Or, the survey 

question may be relatively short, leaving more room for 
responses (e.g., if the average number of responses 
provided was 5) . The total number of available 
responses to a given survey question ( 11 Y 11 ) may also be 

10 identified (see for example, FIGS. 2 and 3). The 

number of "None of these" responses that were made only 
because the appropriate response was hidden when the 
survey question was presented (e.g., the survey 
participant would have selected a hidden response) may 

15 be estimated as follows: 



Hidden = 1 - (X/Y) 



Where : 

Hidden = Percentage of responses 
hidden on average screen 



Where : 

Reallocated = 

Sample * Hidden Reallocated = Number of "None of 

these" responses that would have 
been different if the entire list 
or survey responses was provided 



Sample = Number of times the 
question was presented 



Real = Where : 

Selected - Reallocated 

Selected = Number of time the 
"None of these" response was 
selected 



Real = Number of "None of these" 
responses which should remain 
such 
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[0058] In the illustrative example, the average 
number of survey responses provided for the given 
survey question ("X") is 4 and the total number of 
5 available responses ("Y") is 7. Therefore, the rate at 
which the "None of these" responses were selected only 
because more appropriate responses (e.g., hidden 
responses) were not provided may be estimated as 1 - 
4/7, or 0.429 (e.g., it may be estimated that 43 

10 percent of the time that survey participants selected 

the "None of these" response when presented with a non- 
inclusive list of available responses, a more suitable 
response would have been selected if all available 
responses had been provided) . Because the survey 

15 question was provided 100 times, the Reallocated value 
may be computed as 10 0 x .43, or 43 . It may be 
determined, therefore, that 43 of the 50 "None of 
these" selection should be reallocated to the remaining 
responses (e.g., based on the selection rate of each 

20 response) . Accordingly, 7 of the original 50 "None of 
these" selections should remain such. 
[0059] The raw percent of responses for all 
responses except the "None of these" response may also 
be totaled (in the above example, 78 percent) . The 

2 5 Reallocated value may be distributed among other 

responses according to the frequency distribution of 
the raw percentages. The adjustments (e.g., 
redistribution) to the other responses will be referred 
to as "adders." Adders for each response may be 

30 determined based on the selection rate of each response 
and the probability that a survey participants 
response would have been different had other survey 
response selections been available. The adders may be 



used to reallocate "None of these" selections. For 
example : 



SR n Where: 
An = — *R 



T 



An = Adder for a given response 

SR = Selection rate for the given 
response 

T = Total selection rate for all 
responses other than "None of 
these" response 

R = Number of "None of these" 
responses to be reallocated 



Accordingly, for response 1 of FIG. 6: 



10 Where : 

A\ = * 41 

M 7g Al = Number of selection counts 

to be added to response 1 



For response 2 of FIG. 6: 

7 Where : 

A2 = — * 43 
78 

A2 = Number of selection counts 
to be added to response 2 

[0060] Each of these "Adders" (e.g., Al, A2 , etc.) 
may be added to the observed selection rate for each 
survey response to determine a more accurate estimation 
of the true selection rates for a survey question had 
all available responses for the selected question been 
provided (e.g., if no survey responses were hidden). 



For example, FIG. 7 shows table 700 which shows how the 
adders are used to adjust the initial selection 
rate 702 to correct for "None of these" responses that 
should be reallocated. The adders 704 may be provided 
and added to the initial selection rate 702 for each 
survey response. An adjusted selection count 706 may 
be provided. Adjusted selection count 706 incorporates 
the "None of these" responses reallocated for each 
question. An adjusted selection rate 708 may then be 
determined for each response by calculating the percent 
of the time each response would have been selected if 
all available responses were always provided (e.g., 
using the number of times the survey question was 
presented) . 

[0061] In an alternative embodiment, determining the 
distribution of reallocated "None of these" selections 
may include comparing survey response selections from 
among various survey participants. For example, a 
survey participant who selected the "None of these" 
response to a survey question and who was not shown 
response option #6 from the full list of responses may 
be compared to all other survey participants who did 
see response option #6 when asked the same question. 
These other survey participates may be filtered to 
determine those survey participants that are most 
similar to the survey participant that did not see 
response option #6. The percentage of survey 
participants that selected response option #6 may be 
used to represent a probability that the survey 
participant who did not see option #6 would have 
selected it had the customer seen survey response #6. 
This process may be repeated for each survey response 
that the survey participant did not see. A probability 
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distribution for estimating the "true" intended 
response among all the responses not shown to a survey- 
part icipant may then be provided. 

[0062] It will be understood that, when using either 

approach for selecting survey responses, survey 
responses may initially be selected by a method other 
than those described by the foregoing. For example, 
survey responses may initially be randomly selected, or 
any other suitable method for initially selecting 
survey responses may be used. This allows a suitable 
sample size to be collected for each response before 
using the foregoing systems and methods to select 
responses . 

[0063] A flow chart 800 of illustrative steps that 
may be involved in selecting survey questions to be 
provided to a survey participant in accordance with the 
present invention is shown in FIG. 8. At step 802, 
survey questions, survey responses and survey response 
information provided by survey participants may be 
stored in a storage device, for example application 
server 108. 

[0064] A survey question may be selected using the 
approaches set forth in steps 804 and 806 or steps 808 
and 810. At step 804, the survey response information 
may be used to determine a response variance for each 
survey question. A question's response variance may be 
calculated, for example, using conventional 
mathematical techniques for determining the variance of 
a data set. At step 806, the response variance may be 
used to select a survey question. For example, survey 
questions having a relatively high variance may be 
selected for inclusion in a survey over survey 
questions having a relatively low variance. Or, survey 
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questions may be selected by comparing their variance 
to a threshold variance, and questions having a 
response variance higher than the threshold variance 
may be selected for inclusion in the survey while 
5 survey questions having a variance lower than the 

threshold variance may be excluded from the survey) . 
[0065] Alternatively, at step 808, an inclusion 
value may be determined for each survey question. The 
inclusion value may be determined, for example, using 

10 conditional branching logic, response variance, a 

global inclusion value multiplier, any other suitable 
information, or a combination thereof. At step 810, 
the inclusion value may be used to select a survey 
question from a list of survey questions. For example, 

15 survey questions having a relatively high inclusion 
value may be selected for inclusion in a survey over 
survey questions having a relatively low inclusion 
value. Or, survey questions may be selected by 
comparing their inclusion value to a threshold 

2 0 inclusion value, and questions having an inclusion 

value higher than the threshold inclusion value may be 
selected for inclusion in the survey while survey 
questions having an inclusion value lower than the 
threshold inclusion value may be excluded from the 
25 survey. At step 812, the selected survey question may 
be provided to the survey participant. 

[0066] A flow chart 900 of illustrative steps that 
may be involved in selecting available responses to 
survey questions in accordance with the present 

3 0 invention is shown in FIG. 9. At step 902, survey 

questions, survey responses, and survey response 
information provided by survey participants may be 
stored in a storage device, for example application 
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server 108. At step 904, a survey question is selected 
from a list of survey questions (e.g., in accordance 
with the foregoing systems and methods for selecting 
survey questions) . At step 906, an initial selection 
5 rate for each survey response for the selected survey 
question may be determined using survey response 
information. A initial responses selection rate may be 
determined by dividing the number of times a response 
was previously selected by the number of times the 

10 response was previously presented in a survey. A 
selection rate for a fallback response may also be 
determined at step 906. A fallback response (e.g., 
"None of these") may be used to allow a survey 
participant to indicate that none of the responses 

15 provided are appropriate. 

[0067] At step 908, a reallocated value may be 
determined which may indicate the number of fallback 
responses that would have been different if previously 
hidden responses been presented to previous survey 

2 0 participants. The reallocated value may be, for 

example, the percentage of "None of these" responses 
that may have been different had hidden responses been 
provided. In another suitable approach, the 
reallocated value may be the actual number of "None of 

25 these" selections that may have been different had the 
previously hidden responses been provided (in this 
approach, the reallocated value represents the actual 
number of responses to be reallocated among the 
remaining responses) . 

30 [0068] At step 910, the initial selection rate for 
each question may be adjusted using the reallocated 
values determined at step 908 (e.g., redistributing the 
selected "None of these" responses to the remaining 
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responses) . This may be accomplished by multiplying 
the number of "None of these" responses to be 
reallocated by the initial selection rate determined in 
step 906 and dividing by the total selection rate of 
5 all responses other than the "None of these" response. 
At step 912, survey responses are selected using the 
adjusted selection rate. For example, survey responses 
having lower adjusted selection rates may be selected 
over responses having higher adjusted selection rates 

10 to increase the sample size of those responses. At 
step 912, the selected survey question and responses 
are provided to a survey participant. 
[0069] It will be understood that the orders of 

steps shown in FIGS. 8 and 9 are merely illustrative 

15 and that orders other than those shown may also be 

used. For example, determining initial selection rates 
for stored survey responses (step 906) may occur in 
real time and these determination may be made prior to 
selecting a survey question to be provided (step 904) . 

20 [0070] It will also be understood that numerous 

arrangements for the forgoing systems and methods may 
be implemented. For example, the foregoing systems and 
methods may be incorporated into retail point of sales 
terminals in a physical store so that survey questions 

25 may be provided to survey participants throughout the 
checkout process (e.g., via sales clerks, customer 
service representatives, etc.). A similar approach may 
be used in an online environment, for example when a 
user proceeds to a "checkout" process using an 

3 0 electronic commerce website provided over on the World 
Wide Web. In another suitable approach, the foregoing 
systems and methods may be implemented on a telephone 
or other interactive voice response system, and known 
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technologies for voice synthesis and voice recognition 
may be used to provide survey questions and responses 
and to receive and store survey response information. 
[0071] The foregoing is merely illustrative of the 
5 principles of this invention and various modifications 
may be made by those skilled in the art without 
departing from the scope and spirit of the invention. 
Those skilled in the art will appreciate that the 
present invention may be practiced by other than the 
10 described embodiments, which are presented for purposes 
of illustration and not of limitation, and the present 
invention is limited only by the claims. 



